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ABSTRACT 

This paper discusses the seven types of research tools that have 
potential for advancing knowledge about student services in California Community 
Colleges. The seven tools are the following: literature review, data validation, 
survey research, case study, quasi experiment, meta analysis, and statistical 
modeling. The report gives reasons why each research tool is important. An 
appreciation of their individual qualities as well as their complementary uses in 
various scenarios is instrumental in developing a multi-year plan of research. With 
the use of a hypothetical example, research in outreach for financial aid, the author 
demonstrates the nuances of the tools as they fit into a plan for research. The report 
concludes with these recommendations: (1) documentation and dissemination of research 

is critical for researchers to link studies with other similar studies; (2) people 
need to recognize that research will have increased value if we can use it to help 
program staff in the field which means that program staff needs to have involvement in 
studies even if the staff may lack the experience and duties of a full time 
researcher; and (3) it is important to incorporate the concept of cost in the research 
program. (Contains 22 references and 2 tables.) (MZ) 
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Abstract 

This paper discusses seven types of research tools that have potential for advancing 
knowledge about student services in the California Community Colleges. An 
appreciation of their individual qualities as well as their complementary uses in various 
scenarios are instrumental in developing a multi-year plan of research (or research 
strategy). With the use of a hypothetical example, research in outreach for financial aid, 
we demonstrate the nuances of the tools as they fit into a plan for research. 
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Introduction 



Legislators and policy-makers seem to have two salient information needs when they 
think of public programs. They need to know which program has done a good job (or a 
bad job). This is the need for information regarding program accountability. They also 
need to know how programs work to achieve (or fail to achieve) designated outcomes or 
outputs. Understanding how a program works enables them to make necessary changes 
to programs, to design new programs, or to estimate budgets. This is the need for 
information that explains the process of a program (that is, the causal model behind it). 

Student services in the California community college system represent one area that 
could benefit from research that produces both kinds of information. However, such 
research will require time, resources, and planning to succeed. Besides the size of the 
community college system, the set of outputs (or services) have qualities that make them 
difficult to analyze. Given the magnitude of the research effort that would be needed, it 
is necessary to formulate a long-term plan to coordinate and support the increments of 
research activity that will cumulatively meet the information needs for accountability and 
explanation (or causal analysis). 

An important step in planning a research strategy (or research agenda) is the planning of 
methods to apply in these research efforts. The following paragraphs tackle this planning 
step by discussing various methods that could contribute to the achievement of the 
research strategy or agenda. 
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A Suite of Research Tools to Consider 



A research strategy will naturally consider alternative ways to study a subject area such 
as student services. Any strategy that we suggest should consider at least the seven 
general research tools listed below. 

Literature review 
Data validation 
Survey research 
Case study 
Quasi-experiment 
Meta-analysis 
Statistical modeling 

Obviously, the above list does not name all research tools that researchers could use in 
analyzing student services. But the list names general tools that at this point seem to 
have especially high potential for the advancement of our knowledge in student services. 
Of course, to appreciate any proposed strategy, we need a brief, summary explanation of 
these tools. We provide this below with the aid of language from various references in 
research. 



Literature review: “A systematic survey and interpretation of the research findings (the 
“literature”) on a particular topic, usually designed to prepare for undertaking further 
research on the subject... ’’[Vogt, 1993, p. 130] 

Researchers do literature reviews for several reasons. Most frequently, they do it 
to see what prior studies of the topic have found. An effective review will help a 
researcher to design a new study that will add to the existing knowledge base by 
testing a hypothesis or collecting new information that previously was 
unavailable. Although some researchers may not have much concern about 
testing a new theory, they will still do a literature review to understand how to do 
a study, that is, to learn about which data to collect and which analyses to employ 
(such as various statistical tests). Researchers also do literature reviews because 
they use summaries of their reviews in their reports (or papers) to provide readers 
a context or background to the subject. In some instances, an extensive literature 
review can stand alone as the researcher’s product, especially when policy makers 
or other analysts need just a current summary of prior work on a topic. 
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Data validation. A process for investigating the quality of a set of measurements. 
Generally, this involves some estimation of the amount of error that the set of 
measurements may contain. 

This is a test for the usefulness of the data in terms of their accuracy in 
representing what they are purported to be. This process is necessary to avoid 
erroneous findings that may stem from the analysis of bad data. The analyst may 
test a data file by comparing it with a matching data file (as in large key entry 
operations where the same data are entered twice by different key entry staff). 
When data about the same student differ between the two files, some suspicion 
about the quality of the data may arise. In some instances, staff may check data 
accuracy by auditing source files for their data (i.e., checking electronic records 
against paper records). A difficult, but critical, test in data validation is the 
verification of the theoretical relevance of measurements (which researchers have 
labeled as “construct validity”). Ideally, validation of a data set will precede 
analyses that may use that data set. 



Survey research: “...Quantitative social research in which one systematically asks many 
people the same questions, then records and analyzes their answers...” [Neuman, 2000, 
p.520] and “...relatively systematic, standardized approaches to the collection of 
information on individuals, households, or larger organized entities through the 
questioning of systematically identified samples of individuals.” [Rossi, Wright, & 
Anderson, 1983, p.l] 

This tool probably could benefit the research of any process or element in student 
services. How students, counselors (or other direct service providers), and 
administrators view things, how they behave, or the circumstances in effect (as 
measured by responses to survey questions) could really help explain outcomes in 
student services and reveal areas of opportunity. Obviously, survey data for this 
research will cost so much that we cannot expect the state’s huge management 
information system to collect such specialized data on an ongoing basis. It is 
prudent to expect survey research to occur on an adhoc basis to accommodate the 
specialized data needs of researchers. 



Case study : “Gathering and analyzing data about an individual example as a way of 
studying a broader phenomenon... The case may be an individual, a city, an event, a 
society, or any other possible object of analysis. . .allows more intensive analyses of 
specific empirical details... The disadvantage is that it is hard to use the results to 
generalize to other cases... ’’[Vogt, 1993, p.30]. 

This tool incorporates elements of qualitative research. In the context of this 
paper, a case study approach allows the researcher to do an in-depth analysis of 
one institution, one counselor, one student, or one family. In that analysis, the 
researcher gathers information through interviews, observation, participation, and 
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records (such as documents). Although the researcher may include numeric data 
in the case study, the bulk of the analysis will deal with the interpretation of the 
unique factors or a pattern of things involving the case in order to develop a 
theory (or simply a conclusion) about a process or situation. This tool can 
advance our knowledge about student services explicitly by unveiling factors or 
patterns that researchers are not considering and by verifying prior inferences 
from quantitative analyses. The completion of multiple case studies may also 
provide us with the building blocks for generalizing a unique finding to a broader 
population. Because case study research (like many qualitative methods) takes 
significant time and resources (largely from the extensive field work involved) 
and is not a standard method of research for accountability purposes, the tool has 
historically had limited use. (Yin, 1989) 



Quasi-experiment: “A type of research design for conducting studies in field or real-life 
situations where the researcher may be able to manipulate some independent variables 
but cannot randomly assign subjects to control and experimental groups. . .’’[Vogt, 1993, 
p. 1 84) and “...studies that resemble experiments but are weak on some of the 
characteristics, particularly the allocation of subjects to groups is not under the 
investigator’s control ... [Everitt, 2002, p.306] 

Although scientists agree that the experimental research design provides the 
strongest evidence of a cause and effect relationship, educational researchers have 
traditionally had trouble with that design’s requirement of random assignment of 
people to either a control group or a treatment group. As a result, research on 
causal relationships (typical in program evaluation) has tried to estimate program 
effects with quasi-experiments, a design that lacks random assignment of people 
to the treatment and control groups. The common pre/post analysis is a familiar 
type of quasi-experimental design. (Campbell & Stanley, 1963; Cook & 
Campbell, 1979). 



Meta-analysis: “quantitative procedures for summarizing or integrating the findings 
obtained from a literature review of a subject. Meta-analysis is, strictly speaking, more a 
kind of synthesis than analysis. The meta-analyst uses the results of individual research 
projects on the same topic... as data points for a statistical study of the topic.” [Vogt, 
1993, p. 138] 

This tool has largely come to prominence in the past two decades, especially in 
education and psychology. It basically gives us a statistical way to find a 
summary estimate of some theoretical relationship that multiple independent 
studies, using different data, have investigated in a comparable fashion. It 
essentially is a statistical upgrade of the literature review tool mentioned above. 

In fact, published meta-analyses generally rely upon a literature review as a 
necessary antecedent to the statistical estimate of a summary level of some 
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theoretical relationship. The utility of meta-analysis grows as the number of 
comparable studies grows. Thus, if researchers cannot locate many comparable 
studies to use in this research synthesis (another name for meta-analysis), its value 
in advancing our knowledge about student services or in informing public policy 
is limited. On the other hand, the availability of meta-analysis makes it feasible to 
conduct a series of independent studies, in a piece-meal fashion if necessary, in 
order to get an overall estimate of a program’s effect or of some other theoretical 
relationship. Thus, the potential for using meta-analysis can allow a research 
program to proceed with a series of small studies instead of one super study (that 
may never get funded). Lastly, well-done meta-analyses can clearly have more 
credibility and persuasiveness than the finding from a single study (or a single 
site). (Cooper & Hedges, 1994;Cook, et al., 1992) 



Statistical modeling: .a description of the assumed structure of a set of 

observations. . .[Everitt, 2002, p.247] 

This tool may occur in the use of four other tools (analysis of a quasi-experiment, 
meta-analysis, survey research, or data validation). However, we list it separately 
to highlight that statistical modeling is a critical tool in its own right, with many 
uses that are not part of a causal analysis. For example, statistical modeling (via 
an extrapolation model) can provide a forecast of student needs, a diagnostic tool 
for identifying the most needy students, or an accountability mechanism for 
institutions. The Chancellor’s Office analysis for persistently low transfer 
colleges is a recent example of this tool in a non-causal, accountability context. 
(Bahr, Horn, & Perry) 



Part of this strategy involves an assessment of the need for researchers to collect new data 
(labeled as primary data in research circles) for a study. It would be naive to believe that 
every important variable, that may explain the process and outcomes of student services, 
already exists in some MIS or in some other researcher’s prior research. Nor should we 
expect every population of interest to be included in pre-existing data (labeled as 
secondary data in research circles). Secondary data often have incomplete “coverage” in 
terms of variables and populations. Unfortunately, even where secondary data are 
comprehensive in coverage (as when a special study on the topic has already occurred), 
they may quickly become obsolete. Moreover, until researchers perform data validation 
on secondary data, they should remember the phrase “handle with caution” and consider 
the option of gathering new data from a random sample. 

In this paper, quantitative research may seem to have a monopoly in the allocation of 
focus. The neglect of qualitative methods in social research has long sparked debates 
from advocates and practitioners of qualitative research, and the recent emphasis on 
quantitative research in federal policy (No Child Left Behind) has re-ignited this 
challenge. In any case, the strategy in this article exploits the advantages of qualitative 
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research methods but in the context of case study methods. The strategy downplays any 
heavy reliance upon qualitative research methods because of two major factors: (1) the 
interest of legislators in strong accountability (which usually requires numerical 
comparisons) and (2) the limitations of qualitative findings as stand-alone information for 
policy-making discussions. (Weiss, 1998; Cook, 1997; Rossi & Freeman, 1993). 
Nonetheless, researchers of specific projects may find opportunities to integrate 
qualitative methods by means of the recent approach of mixed methods. (Teddlie & 
Tashakkori, 2003; Weiss, 1998; Cresswell, 1998). 



Now that we have described the tools, we can consider the context of each tool for a 
research strategy/agenda. Three important issues in a research agenda would be the place 
of a tool in a sequence of efforts, the value of a tool for providing program accountability, 
and the value of a tool for explaining causal processes. Conventionally, the largest 
concerns among evaluation researchers have focused upon the goal of accountability 
versus the goal of knowledge (explanatory value in this article), but we place the concern 
of stage in agenda as an important qualifying factor for our planning strategy. 

(Chelmisky, 1997). Figure 1 displays in our judgment how the seven tools stand in terms 
of the three important issues mentioned above. 





Research Tool: 


Stage in Agenda 


Accountability 

Value 


Explanatory Value 
(Causal Argument) 


1 . 


Literature review 


early 


low 


low or medium 


2. 


Data validation 


early 


low to medium 


low 


3. 


Survey research 


middle 


medium 


medium 


4. 


Case study 


early or middle 


low 


medium 


5. 


Quasi-experiment 


late 


high 


high 


6. 


Meta-analysis 


late 


low or medium 


high 


7. 


Statistical modeling 


any 


high 


medium 



Figure 1: A Comparison of Research Tools in a Multi-year Strategy 



Although literature review, case study, and data validation appear in Figure 1 to lack 
special value for either accountability or causal explanation, each of these tools can 
substantially increase the chance of success with the other tools in their resulting value 
for accountability or causal explanation. Meta-analysis has low accountability value in 
Figure 1 because meta-analysis often demonstrates the general linkage of a policy to an 
outcome. Unless the meta-analysis is limited to the institutions or programs for which 
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the public demands accountability (such as studies only about a single state’s 
institutions), the value for accountability purposes is not high. 

Hopefully, a data validation study would precede the use of secondary data, but we 
realize the tentativeness of that expectation, given the pressures that often act to expedite 
analyses. In the same token, we hope researchers include a data validation step within a 
study that uses primary data. From a realistic viewpoint, we should expect an iterative 
process with loops backward or links between tools to signify the interactive behavior 
that a research agenda can have over time. In any case, the sequence with which any of 
the other tools will be used is uncertain, given the unique environment that may exist 
when a particular topic in student services is selected as a research topic. Finally, a 
particular topic may not require more than a couple of these tools, and the 
budget/staffing/administrative environment may not allow more than that. 

This discussion should clarify the selection process for a research tool to address a 
potential topic in student services. To clarify further the possible scenarios for future 
applications of these tools, we turn to how the tools could help in the study of a particular 
part of student services, outreach to increase student usage of financial aid. 



An Example: Outreach Program for Financial Aid 

As a demonstration of how the general strategy may apply to a concrete situation, we can 
use a hypothetical scenario. In this hypothetical scenario, the state is planning to study a 
proposed outreach program that is intended to increase the use of financial aid by target 
populations. To make this example meaningful, we will first review some specific issues 
that distinguish financial aid outreach from other programs so that the link to the choice 
of research tools has more realism. 

The process of outreach generally could follow a basic model like that shown in Figure 2. 
This model is critical in highlighting explicit steps in the outreach process that otherwise 
would resemble the proverbial “black box.” Although a full literature review would 
generate a model that would use extensive documentation, we offer this model based 
upon a cursory extraction of concepts from several disciplines (communications, 
marketing research, and health education). (Simmons, 1990; Littlejohn, 1992; Solomon, 
1994; Fishbein & Ajzen, 1975; Renger, Steinfelt, & Lazarus, 2002) 




Figure 2: A Hypothetical Model for Outreach Process 



One immediate issue that a research agenda would address is the following question: Is 
the purpose of the research program development/design or program accountability? 

This issue will largely determine which research tool among those in Figure 1 has priority 
in a schedule of efforts. 

If there is a priority for accountability, then the kinds of questions for the immediate 
phase of analysis could include the following, for example: 

1. What explicit student-level result should be a program objective? In other words, 
is awareness, acceptance (of message), intent, or behavior (actual receipt of 
financial aid) the proper level of “success?” 

2. How does the program benefit the program’s target group? Does receipt of 
financial aid contribute to student outcomes, such as retention, persistence, 
transfer, employment, or certificates/degrees? 

3. Does the program affect disadvantaged special populations differently from other 
populations? Does a financial aid program benefit individuals who have less need 
than individuals who have more need for aid, thus creating social inequity? 

4. Are resources efficiently used in the program? Is a particular mode of program 
operation or a particular phase in the service process obtaining less output per 
dollar than some other mode or phase? 



In contrast, if priority lies in understanding how the program operates to benefit students, 
then the kinds of questions to address at the start could include the following, for 
example: 

1. Is there a sequence of states or events that promote the use of financial aid? Do 
students need to crystallize educational plans; experience some coursework at the 
community college; appreciate the long-term benefits of financial aid; and accept 
aid’s consequences (either as an obligation or as a grant) before they obtain 
financial aid? 

2. What factors influence student use of financial aid beyond financial need? Do 
things like student aspirations, family context, job market, demographic factors, 
risk aversion attitude, and academic preparation moderate the likelihood of using 
financial aid? 

3. Does a particular channel of communication work better than others? Are word- 
of-mouth from peers, notices in student newspapers, and campus websites equally 
effective in promoting the use of financial aid by target populations? 



4. Does a particular message content and format work better than others? Does a 
simple slogan repeated often work better than a single detailed description of the 
financial aid process? 

5. Does a particular message source work better than others? Do a college financial 
aid officer, a college student leader, and a state official have equal effectiveness 
as a message source? 

6. Does a particular target audience vary on the above factors? Do immigrant 
populations; working students; and disabled students, among others, have 
different reactions to a certain type of outreach effort? 



For each of these questions, one or more research tools may be more effective than the 
other tools. A research strategy should consider the matching of tools to a particular 
question. As a demonstration of this concept, we present one such matching of tools to a 
research question (what factors influence use of financial aid) in Figure 3. The process 
begins with a literature review because the results of this step can improve all ensuing 
steps. Next, we might use the case study tool to study several community college 
students to obtain an in-depth picture of (1) how students learn about financial aid and 
other college-related matters; (2) what their attitudes toward financial aid are; and (3) 
what their college-related behavior patterns are. The information from the case studies 
would then help researchers to test their theories about student beliefs and behaviors with 
the use of survey research and/or quasi-experiments. The survey research would help 
researchers estimate the relative influence of various factors, especially attitudinal ones. 
The quasi-experiments would go further in verifying that certain types of outreach effort 
work at certain levels with certain student populations. The segment for statistical 
modeling would involve the analysis for producing performance indicators and/or so- 
called adjustment performance measures (as in the analysis for persistently low transfer 
colleges). 
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causal analysis phase 



accountability 
analysis phase 



Figure 3: Plan for Factors Influencing Use of Financial Aid 



A different matching of tools could easily arise, given different circumstances. For 
example, we present in Figure 4 an alternative plan for matching tools. If researchers can 
locate a good set of relevant studies (that deal with financial aid outreach), then the 
researchers could use a meta-analysis (or research synthesis) to obtain quantitative 
estimates of relationships (like the “success rate” of a practice in moving students to 
apply for financial aid) in lieu of the detailed data-gathering that would come from the 
case studies, surveys, and quasi-experiments. So the ability to use meta-analysis could 
save money and time, but the researchers’ access to usable studies on the topic is a 
necessary element here. 




causal analysis phase 



accountability 
analysis phase 



Figure 4: Plan for Factors Influencing Use of Financial Aid Via Meta-Analysis 



In concluding this example, we should stress that causal analysis should have more 
emphasis than accountability analysis. Ideally, a long-term research agenda would try to 
encompass both purposes, and the entire set of tools would be applicable from that 
perspective. Ideally, accountability analysis should build upon causal analysis if 
oversight bodies expect performance indicators to have meaning and equity for program 
staff and program administrators. Otherwise, an accountability system will appear as 
arbitrary and capricious meddling to the programs (at least to their staff and 
administrators), and the programs may oppose or resist accountability efforts. So it 
would be helpful if causal analysis preceded accountability analysis in time. However, in 
the real-world environment of policy-making, decision-makers generally prefer 
accountability research over causal research because causal research often takes too long 
to do and often produces seemingly inconclusive or complex findings, in comparison to 
accountability research. 
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Conclusion 



We should recognize the various research tools that could help us study student services. 
That step will help the system to develop a coherent and effective, multi-year strategy for 
developing knowledge about student services. But a system strategy for knowledge 
development can do more than just make appropriate use of research tools. Although the 
following points deserve their own paper, we briefly mention them here to help us realize 
that there are some other things that the system could do to enhance its research efforts in 
student services. These points were also made years ago for the improvement of research 
in clinical psychiatry. (Linden & Wen, 1998) 

1. We need to understand that a piece of research, a study, gains value when we can 
link it to other similar studies. This argues for a system level understanding for 
the concepts of replication in research and of study comparability. A series of 
small studies that replicate an initial study can be both more feasible and more 
valid than a single large study. (Rosnow & Rosenthal, 1984) At the same time, 
the ability to link findings from different studies depends heavily upon the ability 
of researchers to know how similar, or dissimilar, the studies are in terms of their 
definitions of outcomes and their data collection/analysis protocols (such as 
sampling). The documentation and dissemination of research is critical. 

2. We also need to understand that the research will have increased value if we can 
use it to help program staff in the field. This means that program staff needs to 
have involvement in studies even if the staff may lack the experience and duties 
of a full-time researcher. Not only can program staff help in the logistics of 
potential studies, this group can also help researchers recognize important 
practical implications of theories. 

3. Finally, we need to incorporate the concept of cost in the research program. 
Obviously, this need has two significant ramifications that have peeved 
educational researchers. It may divert the focus of research away from student- 
centered processes and towards budgeting and institutional accountability. 

Second, it requires the use of expertise in economics, a field that most educational 
researchers would not claim as their own. Nevertheless, the analysis of benefits 
and costs of programs is an undeniable need in the system, and the Chancellor’s 
Office project on the “real cost of providing quality community college education 
for all students” is just one effort to work in this area. 

Admittedly, the subject of a long-range research plan for student services deserves much 
more discussion and attention than it has received in the past, or in this paper for that 
matter. The stakes are high today and throwing up our hands in despair (because of 
inadequate resources or lack of interest) will not serve the system well during the next 
generation. 
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